Pietro Pollo, Szymon M. Drobniak, Hamed Haselimashhadi, Malgorzata Lagisz, Ayumi Mizuno, Daniel W. A. Noble, Laura A. B. Wilson, Shinichi Nakagawa
Published
August 17, 2025
1 Update
We will update this tutorial when necessary. Readers can access the latest version in our GitHub repository.
If you have any questions, errors or bug reports, please contact Pietro Pollo (pietro_pollo@hotmail.com) or Shinichi Nakagawa (snakagaw@ualberta.ca).
2 Introduction
This online material is a supplement to our paper “Beyond sex differences in mean: meta-analysis of differences in skewness, kurtosis, and correlation”. You will see how to calculate the new effect size statistics we have proposed and how to use them in a meta-analytical model using the metafor package in R.
3 Content
In this online material, we provide details on the simulations we ran to evaluate the effectiveness of our proposed effect size statistics (and their associated sampling error). We also show how to (1) calculate our newly proposed effect sizes (\(\Delta sk\), \(\Delta ku\), \(\Delta Zr\)) and (2) exemplify their use with data from the International Mouse Phenotyping Consortium.
4 Simulations
4.1 Extended Simulation Details
We conducted Monte-Carlo simulations to evaluate bias and variance estimation for our new effect sizes \(\Delta sk\), \(\Delta ku\), and \(\Delta Zr\). For \(\Delta sk\) and \(\Delta ku\) we simulated independent samples for two groups from Pearson distributions with known moments using the rpearson function from the PearsonDS R package (vers. 1.3.2, Becker and Klößner 2025). We conducted two simulations: 1) first by changing skewness between groups that involved moderate departures from normality (group-specific skewness, \(sk \in \{-1, -0.5, 0, 0.5, 1\}\) with kurtosis fixed at 3) and 2) by holding skewness constant (\(sk\) = 0) while manipulating kurtosis, \(ku \in \{2.5, 3, 4, 5, 6\}\). In all cases, we simulated scenarios where: (i) the variance between each group was the same (\(\sigma^2_{2}\) = \(\sigma^2_{1}\) = 1) or different (\(2\sigma^2_{2}\) versus \(\sigma^2_{1}\)); (ii) the mean between the two groups was the same (\(\mu_{2}\) = \(\mu_{1}\) = 0) or different (\(\mu_{2}\) = 5, \(\mu_{1}\) = 0); and (iii) under equal and unequal sample sizes between groups with sample size varying from \(n \in \{10, 20, \dots, 100, 150, 500\}\). We created all unique combinations of the above scenarios resulting in \(s\) = 1200 independent scenarios (when considering each of the 100 scenarios at each sample size, see examples in Section 4.3 and Section 4.4). We estimated \(\Delta sk\) and \(\Delta ku\) for each scenario using formulas for within-group sample skewness with small-sample correction (Eq. 1 in main manuscript) and excess kurtosis with small-sample correction (Eq. 3 in main manuscript) to estimate point estimates. To estimate associated sampling variance for \(\Delta sk\) and \(\Delta ku\) we used the analytical variance estimators derived here and an associated re-sampling (jackknife) approach to compute group sampling variances separately followed by pooling. Importantly, our simulations assume no correlation between groups.
For \(\Delta Zr\) simulations, we simulated two groups each containing two variables with known correlations within each group. For \(\Delta Zr\) we drew bivariate normal data with target within-group correlations \(r \in \{-0.8, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8\}\) using the mvnorm function in the MASS package of R (vers. 7.3.61, Venables and Ripley 2002). Marginals were standard normal and group sizes varied from \(n \in \{10, 20, \dots, 100, 150, 500\}\). Again, we created all unique combinations of scenarios resulting in \(s\) = 768 unique scenarios. We estimated \(\Delta Zr\) using Fisher’s Z transformation \(Zr\) and calculating \(\Delta Zr\) as the difference of \(Zr\) across groups (Eqs. 9–11 in main manuscript). Sampling variance for \(\Delta Zr\) was approximated used the standard analytic variance \(\frac{1}{n-3}\) per group (summed; Eq. 10 in main manuscript) and a jackknife approach. Again, we assumed no correlation between our groups.
Across all simulations we conducted 2,500 replicates of each scenario. Performance metrics were (a) bias of the point estimator (mean estimate minus truth) (Equation 1), (b) relative bias of the sampling-variance estimator (Equation 2), and (c) Monte-Carlo standard errors (MCSEs) (Equation 3).
where \(\theta\) is the true overall mean effect (i.e., the true \(\Delta sk\), \(\Delta ku\), and \(\Delta Zr\) between groups), and \(\hat{\theta}_i\) is the effect estimate from simulation \(i\).
To understand bias in the sampling variance, we computed the relative bias for the effect measure as:
where \(\hat{\theta}^2\) is the estimated sampling variance for the effect measure and \(\mathbb{E}[\hat{SV}]\) is the expected value of the sampling variance. Relative bias can be calculated using different combinations of estimates. For example, we can use \(\hat{\theta}^2\) as the demoninator from a point estimate taken from the small sample correction for \(\Delta sk\), \(\Delta ku\), or \(\Delta Zr\) or from the jackknife estimate. Alternatively, we can use the jackknife estimate for \(\mathbb{E}[\hat{SV}]\) across each simulation. We created each of the four combinations for our performance measures. For each of our performance measures we also computed the Monte Carlo Standard Errors (MCSE) of the estimated bias with
Our tutorial uses R statistical software and existing R packages, which you will first need to download and install.
If the packages are archived in CRAN, use install.packages() to install them. For example, to install the metafor , you can execute install.packages("metafor") in the console (bottom left pane of R Studio).
Version information of each package is listed at the end of this tutorial.
We also provide some additional helper functions to calculate effect sizes, process data, and visualise our results. The most straightforward way to use these custom functions is to run the code chunk below. Alternatively, paste the code into the console and hit Enter to have R ‘learn’ these custom functions.
If you want to use these custom functions in your own data, you will need to change the variable names according to your own data (check out the R code and you will see what we mean).
We then use the data from multiple phenotyping centres and mice strains to calculate average effect sizes (\(\Delta sk\), \(\Delta ku\), and \(\Delta Zr\)).
map2_dfr(.x =c("fat_mass","glucose"),.y =c("heart_weight","total_cholesterol"),.f = process.cor_effects) %>%mutate(relationship =rep(c("fat mass and heart weight","glucose and total cholesterol"),each =7)) %>%relocate(relationship) %>%datatable(.,extensions ="Buttons",rownames =FALSE)## Warning: Using an external vector in selections was deprecated in tidyselect 1.1.0.## ℹ Please use `all_of()` or `any_of()` instead.## # Was:## data %>% select(chosen_trait_2)## ## # Now:## data %>% select(all_of(chosen_trait_2))## ## See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.